📚 Week 6 · Unit V · Lecture 17
Monitoring in DevOps &
Log Aggregation

Entering the final phase of the DevOps lifecycle: How we achieve complete observability into our infrastructure, applications, and logs to prevent downtime and improve performance.

Dr. Mohsin Furkh DarSchool of Computer Sciences
DateMon, 13 Jul 2026 · 11:00 AM – 12:00 PM
ProgrammeBTech CSE – Summer Semester
Today's Agenda
1
Unit V Welcome — The "Monitor" Phase
2
Why Monitoring is Critical in DevOps
3
The Three Pillars of Observability
4
Infrastructure vs. Application Monitoring (APM)
5
Log Aggregation & The ELK Stack
6
Alerting & Incident Management
The DevOps Loop
Completing the Infinity Loop

We have successfully planned, coded (Git), built, tested, and deployed. Now, the software is live in Production. We must enter the final phase: Monitor.

Plan
Code
Build
Test
Release
Deploy
Operate
MonitorWe are here
The Purpose of Monitoring
  • To ensure the system is available and performing well.
  • To detect anomalies before the user notices them.
  • To gather data that feeds back into the "Plan" phase for the next sprint.
The Cost of Blindness
  • Without monitoring, the first time you hear about an outage is when angry users tweet about it.
  • Debugging takes days instead of minutes.
  • Capacity planning becomes guesswork.
Philosophy
Reactive vs. Proactive Monitoring

If you have to log into a server to figure out why an application is slow, your monitoring is broken. Monitoring should tell you the 'why' before you even open a terminal.

Reactive (Traditional IT)

Waiting for a server to crash or a user to submit a support ticket. The team scrambles to find the root cause under immense pressure. It is stressful, slow, and damages business reputation.

Proactive (DevOps)

Tracking trends (e.g., "Memory usage is growing 1% per hour"). The system alerts the team automatically so they can fix the memory leak hours before the server actually crashes.

Observability
The Three Pillars of Observability

Monitoring tells you if a system is working. Observability lets you ask why it isn't working. True observability relies on three types of telemetry data.

📈
1. Metrics
What: Numeric representations of data measured over intervals of time (e.g., CPU %, Error Rate).
Why: Great for dashboards and triggering alerts. Cheap to store.
📜
2. Logs
What: Immutable, time-stamped records of discrete events that happened over time.
Why: Provides the fine-grained context needed for deep debugging.
🕸️
3. Traces
What: The end-to-end journey of a single user request as it moves across distributed microservices.
Why: Pinpoints exactly which microservice is causing the slowdown.
Scope
Monitoring the Entire Stack

In modern architectures, you cannot just monitor the server. You must monitor every layer of the stack, from the hardware up to the user's browser.

4. User Experience (RUM / Synthetics)
Browser, Mobile App, Click Rates
3. Application Performance (APM)
Code Execution, DB Queries, Endpoints
2. Infrastructure / Orchestration
Kubernetes, Docker Containers, Pods
1. Hardware / Network
CPU, RAM, Disk I/O, Network Latency
💡 The "Watermelon" Effect

If you only monitor Layer 1 (Infrastructure), your dashboard might be entirely green (CPU is fine, memory is fine). But the application (Layer 3) might be throwing 500 errors. It looks green on the outside, but it's red on the inside. You must monitor all layers.

Layer 1 & 2
Infrastructure Monitoring
🖥️
Infrastructure monitoring ensures the underlying foundation of your application is healthy. It tracks physical servers, virtual machines, network switches, and container orchestrators (like Kubernetes).
Key Metrics Tracked
  • Compute: CPU utilization, Load Average.
  • Memory: RAM usage, Swap space.
  • Storage: Disk space available, Read/Write IOPS.
  • Network: Bandwidth, Packet loss, Latency.
Popular Tools
  • Prometheus: The industry standard for Kubernetes monitoring (pull-based metric scraper).
  • Grafana: Visualization tool usually paired with Prometheus.
  • Datadog / Nagios / Zabbix: Full-suite commercial/legacy tools.
Layer 3
Application Performance Monitoring (APM)

APM looks inside the code as it runs. It tells you exactly which function, database query, or API call is slowing down the application.

⏱️
Response Times
Tracks the latency of every API endpoint. Identifies which specific routes (e.g., /checkout) are performing poorly.
🗄️
Database Profiling
Identifies exact SQL queries that are taking too long. Highlights missing database indexes or N+1 query problems.
🐛
Error Tracking
Captures stack traces the moment a user encounters a 500 error, allowing developers to see the exact line of code that failed.
🔧
Popular APM Tools: New Relic, AppDynamics, Dynatrace, Datadog APM, and OpenTelemetry (for open-source distributed tracing).
The Challenge
The Microservices Logging Problem

In the old days of monoliths, you had one server. If the app crashed, you SSH'd into the server and ran tail -f /var/log/app.log. Easy.

🔥
In modern DevOps (Microservices & Kubernetes), a single user request might touch 15 different containers spread across 5 different servers. If it fails, how do you know which container's log file to check? SSH is no longer possible.
The Nightmare
  • Containers are ephemeral — when they die, their local logs are deleted forever.
  • Searching for a specific transaction ID across 50 servers manually takes hours.
  • Different apps use different log formats.
The Solution: Log Aggregation
  • Every container streams its logs to a centralized database in real-time.
  • Logs are parsed, indexed, and made searchable.
  • Engineers search one central dashboard instead of touching servers.
Log Aggregation
The ELK (or EFK) Stack

The industry standard for open-source log aggregation is the ELK stack, maintained by Elastic.

📥
L: Logstash (or Fluentd)
The Shipper/Processor: An agent running on every server. It collects local logs, parses them, transforms them into JSON, and ships them to the database.
🗄️
E: Elasticsearch
The Database: A highly scalable, distributed NoSQL search engine. It indexes the JSON logs, allowing you to search terabytes of data in milliseconds.
📊
K: Kibana
The UI: The visualization layer. A web dashboard where developers can query logs, create pie charts of errors, and monitor log velocity.
ℹ️
EFK vs ELK: In modern Kubernetes environments, Logstash is heavy and memory-intensive. It is often replaced by Fluentd (or Fluent Bit), creating the EFK Stack. Other commercial alternatives include Splunk and Datadog Logs.
Distributed Systems
Distributed Tracing

Even with centralized logs, following a single user request through a maze of microservices is hard. Tracing solves this.

How Tracing Works
  • When a request enters the system (e.g., API Gateway), it is assigned a unique Trace ID.
  • Every microservice that handles the request passes this Trace ID to the next service (via HTTP headers).
  • Each service's work is recorded as a Span (with start and end times).
The Result
  • You get a visual "waterfall" graph of the request.
  • You can see: "The request took 2.0s total. Service A took 0.1s, Service B took 0.1s, and Service C took 1.8s waiting on the Database."
  • Tools: Jaeger, Zipkin, OpenTelemetry.
Actionable Data
Alerting & Incident Management

Collecting metrics and logs is useless if nobody looks at them. Alerting bridges the gap between the system and the humans.

🔔
Smart Thresholds
Instead of static alerts ("Alert if CPU > 90%"), modern tools use anomaly detection ("Alert if CPU is 3 standard deviations higher than normal for a Tuesday morning").
📱
On-Call Routing (PagerDuty)
When a critical alert fires at 3 AM, tools like PagerDuty check the team schedule and call/page the specific engineer who is currently on-call.
💬
ChatOps
Non-critical alerts are routed directly to team Slack/Teams channels. Teams can acknowledge alerts and run diagnostic commands directly from the chat interface.
⚠️ Alert Fatigue

A major anti-pattern in DevOps is configuring too many minor alerts. If the on-call engineer's phone buzzes 50 times a day for non-issues, they will start ignoring it. When a real outage happens, it gets missed. Rule: Only alert humans if human intervention is required right now.

Summary
Key Takeaways — Lecture 17

Monitoring transforms blind operations into proactive, data-driven DevOps.

01
Proactive vs ReactiveMonitoring should tell you about a problem (and why it's happening) before the user notices it.
02
Three PillarsObservability relies on Metrics (numbers), Logs (events), and Traces (request journeys).
03
Full Stack MonitoringYou must monitor Infrastructure (servers/network), Application Performance (code/DBs), and End-User Experience.
04
Log AggregationIn distributed systems, logs must be centralized. The ELK/EFK stack (Elasticsearch, Logstash/Fluentd, Kibana) is the standard.
05
TracingTrace IDs follow a request across microservices to generate a waterfall graph of latency.
06
Next: Lecture 18The final lecture! We cover Security in DevOps (DevSecOps), integrating security scanning into CI/CD, and a course wrap-up.
🎯
Exam tip: Know the Three Pillars of Observability. Be able to define the ELK acronym and what each letter does (E: Database, L: Shipper/Parser, K: Dashboard). Understand why Log Aggregation is mandatory for microservices.